Information Spectrum Approach to Overflow 
Probability of Variable-Length Codes with 
Conditional Cost Function 
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Abstract — Lossless variable-length source coding 
with unequal cost function is considered for general 
sources. In this problem, the codeword cost instead 
of codeword length is important. The inflmum of av- 
erage codeword cost has already been determined for 
general sources. We consider the overflow probability of 
codeword cost and determine the infimum of achievable 
overflow threshold. Our analysis is on the basis of 
information-spectrum methods and hence valid through 
the general source. 

I. Introduction 

Lossless variable-length coding problem is quite impor- 
tant not only from the theoretical viewpoint but also from 
the viewpoint of its practical applications. The most fun- 
damental criterion to evaluate the performance of variable- 
length codes is the average codeword length. The overflow 
probability of codeword length is one of other criteria, 
which denotes the probability of codeword length per 
symbol being above a threshold R > 0. The overflow prob- 
ability has been investigated in several contexts. Merhav 
PQ has determined the optimal exponent of the overflow 
probability given R for unifilar sources. 

On the other hand, as is well known, if we impose 
unequal costs on code symbols, we have to consider the 
codeword cost, instead of codeword length. The average 
codeword cost, which is a generalization of the average 
codeword length, have been first analyzed by Shannon 
[2]. Karp [3] has given the variable- length code, which 
minimizes the average codeword cost. The infimum of the 
average codeword cost has been determined by Krause [3] 
for i.i.d. sources and extended to general sources by Han 
and Uchida [5]. Among others, Uchida and Han [B] have 
proposed the overflow probability of codeword cost. They 
have considered the overflow probability as the probability 
of codeword cost per symbol being above a threshold. 
Then, they have shown the infimum of achievable thresh- 
old, where achievable threshold means that there exists a 
code whose overflow probability of codeword cost decreases 
with given exponent r. 
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In this paper, we also deal with the overflow proba- 
bility of codeword cost. In particular, we consider the 
e-achievable threshold, which means that there exists a 
variable-length code, whose overflow probability is smaller 
than or equal to e. Then, we first determine the infimum 
of e-achievable threshold for general sources. Next, we 
consider the second-order achievable threshold for general 
sources. The finer evaluation of the achievable rates, called 
the second-order achievable rates, have been analyzed in 
several contexts [7] -[10]. Analogously to these settings, 
we define the second-order achievable threshold on the 
overflow probability and derive the infimum of the second- 
order achievable threshold. Nomura and Matsushima [TT] 
have first derived this quantity in the case that the cost 
function is assumed to be memoryless. In this paper, we 
treat the conditional cost function to generalize this result. 
As a result, the similar analysis with which is on the 
basis of the information- spectrum methods, is also efficient 
in this case. These results show that the information- 
spectrum methods are still effective in analyses of the 
overflow probability in variable-length coding problem. 

II. Preliminaries 
A. Variable-length codes with cost for general source 
The general source is defined as an infinite sequence 



X = 



{a" = (xf 



] xi n) ■ ■ ■ X, 



•for, 



of n-dimensional random variables X n , where each com- 
ponent random variable -X.- takes values in a countable 
set X. 

The variable-length codes are characterized as follows. 
Let 

ip n : X n -> W, Vn : {^„(x)} xe *„ -> X n , 

be a variable-length encoder and a decoder, respectively, 
where U = {1, 2, • • • , K} is called the code alphabet and 
U* is the set of all finite-length strings over U excluding 
the null string. 

We consider the situation that there are unequal costs 
on code symbols. Let us define the cost function c :U* — s- 



(0, +oo) considered in this paper. The cost c(u ) of a 
sequence u £ U l is defined by 

i 

c(u l ) = J2<u i \u\- 1 ), 

i=l 

where c(tti|u^ _1 ) is a conditional cost of Ui given u^ 1 such 
that < c^K" 1 ) < oo (Vi,Vu, £ U^u^ 1 e U 1 - 1 ). The 
conditional cost capacity a c (?4 _1 ) given u^" 1 is defined by 
the positive unique root a of the equation 

In this paper, we assume that the conditional cost capacity 
a c (tt^ _1 ) is independent on it^ , more exactly, a c (w^~ 1 ) = 
a holds for all u 1 ^ 1 <E W~ x . 

Such a class of cost function has been first considered 
in [T^]. Han and Uchida [5] also have treated this type of 
cost function. We denote 

Cmax = max c(u i |up 1 ) 
i,ui£U,u\ 1 eu i - 1 

for short. The variable-length codes considered in this 
paper, satisfies the prefix condition: 

V ^-"cC( V „(x)) < L ^ 

Throughout this paper, the logarithm is taken to the base 
K. 

B. Overflow Probability of Codeword Cost 

The overflow probability of codeword length is defined 
as follows: 

Definition 2.1: Given a threshold R, the overflow prob- 
ability of the variable-length encoder ip n is defined by 

e n (<p n ,R)=PrlU(<p(X n )) 

where l(-) denotes the length of a string. 

In this paper, we generalize the above overflow probabil- 
ity not only to the case for unequal costs on code symbols 
but also for finer evaluation of the overflow probability. To 
this end, we consider the overflow probability of codeword 
cost as follows: 

Definition 2.2: Given some sequence {??n}^Li, where 
< rj n < oo for each n = 1, 2, • • • , the overflow probability 
of the variable-length encoder (p n is defined by 

£n(fn,Vn) = Pr{c(^„(X")) > T) n } . (3) 

Since {?7n}^Li is an arbitrary sequence, the above def- 
inition is general. In particular, we shall consider the 
following two types of overflow probability in this paper: 

1) n n = nR, 

'2 n n = na + y/nL. 



Remark 2.1: If we set rj n = nR for all n = 1, 2, • • • , the 
overflow probability can be written as 

e n (<p n) nR) =Pr|ic(^ n (X")) > i?J . 

Thus, in the case that rj n = nR, the overflow probability 
defined by ((3]) means the probability that the codeword 
cost per symbol exceeds some constant R. This is a 
natural extension of the overflow probability of codeword 
length analyzed in [I] or [6] to the overflow probability of 
codeword cost. 

On the other hand, the finer evaluation of the achievable 
rates, called the second-order achievable rates, has been 
investigated in several contexts [7] -[10]. Analogously to 
their analyses, we evaluate the overflow probability in the 
second-order sense. To do so, we consider the second case: 
rj n = na + L^/n for all n = 1, 2, • • • . Hereafter, if we con- 
sider the overflow probability in the case r\ n = na + L^/n, 
we call it the second-order overflow probability given a in 
this paper, while in the first case it is called the first-order 
overflow probability. The second-order overflow probability 
given a of variable-length encoder Lp n with threshold L is 
written as 

e n (ip n ,na + Ly/n) = Pr j— > L j ■ 

In the first case, we are interested in the infimum of 
threshold R that we can achieve. 

Definition 2.3: Given < e < 1, R is called an 
e-achievable threshold for the source if there exists a 
variable-length encoder tp n such that 

lim sup e n (ip n , nR) < e. 

n— ¥ oo 

Definition 2-4-' 

R(e\X) = inf {R\R is an e-achievable threshold} . 

Also, in the analysis of second-order overflow probabil- 
ity, we define the achievability: 

Definition 2.5: Given < e < 1 and < a < oo, L is 
called an (e, a)-achievable threshold for the source, if there 
exists a variable-length encoder (p n such that 

lim sup e n {tp n , na + L\fn) < e . 

n— >oo 

Definition 2.6: 
L(e,a|X) = inf {L\L is an (e, a)-achievable threshold} . 
III. Key Lemmas 

We show two lemmas that have important roles to derive 
our main theorems. These lemmas are derived using the 
information- spectrum method and hence valid through the 
general source. 

Lemma 3.1: For any general sources X and any se- 
quence of positive number {^n}^!, there exists a variable- 
length encoder ip n that satisfies 

e n ^n,Vn) < Pi- {z n Px^X n )<K- a ^} + z n K a ^+\ 



for n = 1, 2, • • ■ , where {zn}^^ is a given sequence of an because log 2 < 1 holds. Therefore, we have proved the 

arbitrary number satisfying z\ > for i = 1, 2, • • • and a c lemma. ■ 

denotes the cost capacity defined in (pj. Lemma 3.2: For any variable- length code and any se- 

Proof: We use the code proposed by Han and Uchida quence {T] n }n=i, it holds that 
[5]. Then, from the property of the code, it holds that 

c«(x)) < log P xn (x) + + c max , (4) 

torn = 1,2, ••• where {z n }£° =1 is a given sequence of an 

tor all n = 1,2, where denotes the encoder of , ., , ,. c ■ . n r • 1 o 

. iii arbitrary number satisfying Zj > U tor i = 1, 2, • • • . 

the code. Furthermore, we set the decoder as the inverse . . . , . 

, t , , , . , , j , , Proof: Let C9„ be an encoder of variable-length code, 

mapping of ip n . Note that the code is a uniquely decodable «t 

code for general sources with countably infinite source 

alphabet . B n = {xe X n \P X n (x) < z n K~ a ^ } , 

Next, we shall evaluate the overflow probability of this 

code - Set S„ = {xe X n |cfa n (x)) > Vn }. (8) 

A„ = {x e A" 1 |*„P X n(X") < } , 

S n = {xG^|c«(x)) >77 „}. 

Then, the overflow probability of this code is given by Pr {Px n (X n ) < ^n-K" Qc,? "} 

e„«,7fc0 = Pr{X«e-S„} = ]Tp^(x) = ^ Px»(x)+ ]T Px„(x) 

< > Px» (x) + > Px- (x) 



Then, we have 



xGS„ xG-B„nS£ 



xeS„nA- <£n{fn,Vn)+ ^ P X' l ( X )- ( 9 ) 

where A c denotes the complement set of the set A. xes„nS„ 

Since Q holds, for Vx e S n , we have 0n the Qther handi for Vx g ^ it holds that 

Px-(x) <z„^- Q ^". 



1 , x log 2 

log Px" (x) H h Cmax > Vn- 



Thus, we have 



Thus, we have 

P x «(x) < #-«<=('7«- c -« )+ lo g 2 ; 

for Vx e 5„. Substituting the above inequality into ©, Pr{P X "(X") < z n K " c? '"} 

we have < £n(¥>n,r?n) + z nK~ acV 



< Pr {X™ e A n } + K- a "^-c ma *)+i°& 2 



xeB„nS£ 

< £n(^n,r?„) + \B n n S£| z„i^"". (10) 



xeS " ni » Here, from @, we have 

= Pr{X"e A„} + |5„n^|^- Q ^""- c — ) +log2 . (6) 



Here, from the definition of A n , for Vx £ A c n , it holds that 



Px»(x) > A aCn \ > J2 K~ acVn = \S c n \K- a ^ 



Thus, we have 



7>--Q c r)„ u-aar\ n This mean that 

i> £i^x)> = KI*L— ■ 



xe^ xeA £ *» ~" |P„n^| < |^| < X Q ^" (11) 



Hence, substituting (fTTj) into (fT0|) . we have 



This mean that 

\S n nA c n \<\A c n \<z n K a ^. (7) 
Substituting © into ©, we have Pr {Px™ {X n ) < z n K~^ } < T7„) + 7^ QcT? " z^i^-"^" 

rin) < Pr {X™ G A n } + z n K ^nn K -^{v n -c max )+io g2 = £„(¥>„, r? n ) + z„. 

_ ri ^yv t /i n j- -f z n j\ , Therefore, we have proved the lemma. I 



IV. Infimum OF £- achievable threshold 

In this section, we determine J?(e|X) for general sources. 
Before showing the theorem, we define the function F(R) 
as: 

1 



F(R) = f limsupPr I — !— log 



> R 



na c °Pxn(I») 

The following theorem is one of our main results: 
Theorem 4.1: For < Ve < 1, it holds that 

i?(e|X) = w£{R\F(R) <e}. 

Proof: The proof consists of two parts. 
(Direct Part) Let Rq be as 

Ro = m£{R\F(R) <e}, 

for short. Then, in this part we prove that 

R(e\X) <i? +7, 

for any 7 > by showing that Rq is an e- achievable 
overflow threshold for the source. Let r] n be as rj n = 
n(Ro + 7), then from Lemma [3.11 there exists a variable- 
length encoder ip n that satisfies 

E n ((p n ,nR Q ) 

< Pr { z n P x ~ (X n ) < K- na ^ Ro+ ^ } + z n K a " c ^ +l , 

for n — 1, 2, ■ • • . Thus, we have 

f 1 1 logz„ 

e n {<p n ,nR ) <Pr< log- — — — - > i? +7H 

[na c Px*i(X n ) na c 

+ z n K a ^+\ 

for n = 1, 2, • • • . Notice that z n > is an arbitrary number. 
Set z n = X _v ^ 7 , then we have 

f 1 1 x/n"i 

s n (<p n ,nR ) < Pri log——— > R +~f- x 

[na c P X n(X n ) na c 



K 



-%/n7+ct c c„ 



<Pr( — log ) >R a 
{na c P x <-(X n ) 

for sufficiently large n, because 7 > ^ a 
Thus, since a c and c max are positive constants, by taking 
lim sup,^^ , we have 

lim sup e n (ip n , nR ) 



as n — > 00. 



< lim sup Pr I — log ) . > Rq 

n^ao y na c P X ™ (X n ) 

Hence, from the definition of Rq we have 
lim sup e n (tp n ,nRo) < e. 

n— >-oo 

Therefore, the direct part has been proved. 

(Converse Part) 

Assuming that R\ satisfying 

Ri < inf {R \F{R) < e} , 



is an e-achievable overflow threshold. Then we shall show 
a contradiction. 

Let r\ n be as rj n = nR\ . Then, from Lemma 13.21 for any 
sequence {z n }^i ( z% > i — 1, 2, • • • ) and any variable- 
length encoder we have 

£„(¥>„, >Pr{P x ~(X n ) < z n K~ na ^} - z n , 

for n = 1, 2, • • ■ . Thus, for any variable-length encoder it 
holds that for each n = 1, 2, • • ■ 



e n ((p n ,nRi)>PT 



1 



log 



1 



>Ri- 



log z n 



Set z n = K ™ 7 , where 7 > is a small constant that 
satisfies 

R 1 +j<ini{R\F(R)<e}. (13) 



Since we assume that (I12p holds, it is obvious that there 
exists 7 > that satisfies the above inequality. Then, we 
have 



E n (tp n ,nRi) > Pr j — log ) , 

for any variable-length encoder. Hence, we have 
lim sup e n (tp n ,nRi) 

n— ^00 

> lim sup Pr { - log — — \- — > R x + 7 i > e, 

n->oo [n P x ™{X n ) J 

where the last inequality is derived from (|13p and the 
definition of F(R). This is a contradiction and the converse 
part has been proved. ■ 
Remark J^.l: In the analysis of e-fixed-length source 
coding for general sources, the function 



F(R) = lim sup Pr 



1 



log 



1 



> R 



P X n(Xn) 

is used in the case that a c = 1 [13] (see, also [HI Theorem 
1.6.1]). This suggests a deep relationship between the 
overflow probability of codeword length in variable-length 
coding and the error probability in fixed-length coding. 
The similar relationship in the second- order case is also 
clarified by Theorem 15.11 in the following section and [8j 
Theorem 3]. 

From the above theorem, we can show a corollary. Before 
describing the corollary, we define the spectral sup-entropy 
rate [Hjf : 

H(X) d = p- lim sup - log ) , . 

Then, the following corollary holds. 
Corollary J^.l: 



i?(0|X) = —H(X) 

a c 



(14) 



(12) 



For any sequence {Zn}^ = i of real- valued random variables, 
we define the limit superior in probability of {^n}JJ° =1 by 
p-limsupyj^^ Z n = inf {/3| lim Pr{Z„ > 13} = 0} (cf. P3|) . 



V. Infimum of (e, a)- achievable threshold 



theorem) it holds that 



So far, we have considered the first- order achievable 
threshold. In this section, we consider the second- order 
achievability. In the second-order case, the infimum (s,a)- 
achievable overflow threshold for general sources is also 
determined by using Lemma 13. II and Lemma 13.21 

We define the function F a (R) given a as follows, which 
is correspondence with the function F(R) in the first-order xhus we have 
case. 



hm P Az^4Et^l<u 



j?r T \ de{ v r> J - \ogPx^{X n ) - na c a v 
r a (L) = hmsupPr< j= > L 



Then, we have 

Theorem 5.1: For < Ve < 1, it holds that 

L(e,a\X)=mf{L\F a (L)<e}. 

Proof: The proof is similar to the proof of Theorem 
14.11 Hence, we omit the proof. ■ 

The above theorem is valid for general sources. Hence 
Thcorcm l5.1l is a quite general result. However, in general 
to compute the function L (e, a|X) is hard. Next, we 
consider a simple case such as an i.i.d. source and we 
address the above quantity specifically. 

For an i.i.d. source, from Corollarv l4.ll we are interested 



in LU,±H{X) 



XJ . To specify this quantity for an 
i.i.d. source, we need to introduce the variance of self- 
information as follows: 

a 2 (X) d ^ E (- log P X (X) - H(X)) 2 . 

In this paper, we assume that the above variance exists. 
Then, from Theorem 15 . 1 1 we obtain the following theorem. 

Theorem 5.2: For any i.i.d. source, it holds that 



L\ e,—H(X) 



1 



X )=-^(X)<t>- 1 (l-s), 



where $ _1 denotes a inverse function of $ and <fr(T) is 
the Gaussian cumulative distribution function with mean 
and variance 1, that is, $(T) is given by 



[ T 1 


" y 2 ' 


/ "7= exp 




l-oo V27T 


2 



dy. 



Proof: From the definition of F a {L), we have 



(15) 



Fi 



H{X)/ 




logP x ^X n )~nH(X) 
s/na c 

■logP X n{X n ) - H(X) 



> L 



> 



On the other hand, since we consider the i.i.d. source, 
from the asymptotic normality (due to the central limit 



Thus, l(s, ±H(X) 
1 



1 



exp 



X ) is given by 



exp 



2 



2 



dy. 



dy. 



L[e,—H(X) 



inf < L 



Since $ 



inf <^ L 




dy <e 



, is a continuous function and monoton- 

ically increases as L increases, we have 



L 



*- x (l-e). 



Therefore, the proof has been completed. ■ 

VI. Conclusion 

We have so far dealt with the overflow probability 
of variable-length coding with cost function for general 
sources. The overflow probability is important not only 
from the theoretical viewpoint but also from the engineer- 
ing point of view. However, there is few research on the 
overflow probability of variable-length coding for general 
sources, even though the equal cost function is assumed. 
Hence, our attempt to analyze the overflow probability is 
meaningful. 

We have revealed that, as shown in the proofs of 
the present paper, the information- spectrum approach is 
substantial in the analysis of the overflow probability of 
variable-length coding with cost function. An application 
to more general cost function is a future work. 

Finally, as described in Remark 14.11 there is a deep 
relationship between the overflow probability of codeword 
length in variable-length coding and the error probability 
in fixed-length coding. It is an interesting problem to con- 
sider the relation of the overflow probability of codeword 
cost to the fixed-length coding. 
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